Focal plane array, (FPA) sensors are widely used in visible-light and infrared imaging systems. More particularly, FPA's have been widely used in military applications, environmental monitoring, scientific instrumentation, and medical imaging applications due to their sensitivity and low cost. Most recently research has focused on embedding powerful image/signal processing capabilities into FPA sensors. An FPA sensor comprises a two-dimensional array of photodetectors placed in the focal plane of an imaging lens. Individual detectors within the array may perform well, but the overall performance of the array is strongly affected by the lack of uniformity in the responses of all the detectors taken together. The non-uniformity of the responses of the overall array is especially severe for infrared FPA's.
From a signal processing perspective, this non-uniformity problem can be restated as how to automatically remove fixed-pattern noise at each pixel location. The FPA sensors are modeled as having fixed (or static) pattern noise superimposed on a true (i.e., noise free) image. The fixed pattern noise is attributed to spatial non-uniformity in the photo-response (i.e., the conversion of photons to electrons) of individual detectors in an array of pixels which constitute the FPA. The response is generally characterized by a linear model:zt(x,y)=gt(x,y)·st(x,y)+bt(x,y)+N(x,y),   (1)where N(x,y) is the random noise, zt(x,y) is the observed scene value for a pixel at position (x,y) in an array of pixels (image) that are modeled as being arranged in a rectangular coordinate grid (x,y) at time t, st(x,y) is the true scene value (e.g., irradiance collected by the detector) at time t, gt(x,y) is the gain of a pixel at position (x,y) and time t, and bt(x,y) is the offset of a pixel at position (x,y) at time t. gt(x,y) can also refer to as a gain image associated with noise affecting the array of pixels, and b(x,y,) the offset image of pixels associated with noise. Generally speaking, gain and offset are both a function of time, as they drift slowly along (with temperature change. One key assumption of this model is that gt(x,y) and bt(x,y) change slowly, i.e., they are constant during the period used for algorithms to recover st(x,y). As a result, the time index for these parameters are dropped hereinafter. The task of non-uniformity correction (NUC) algorithms is to obtain st(x,y) via estimating the parameters g(x,y) and b(x,y) from observed zt(x,y).
Prior art non-uniformity correction (NUC) algorithms can be grouped into two main categories: 1) calibration methods that rely on calibrating an FPA with distinct sources, e.g., distinct temperature sources in long wave infrared (LWIR), and 2) scene-based methods that require no calibration. Prior art calibration methods include two-point and one-point non-uniformity correction (NUC) techniques. Two-point NUC solves for the unknowns g(x,y) and b(x,y) for all the (x,y) pixels in Equation 1 by processing two images taken of two distinct sources e.g., two uniform heat sources in an infrared imaging system (i.e., a “hot” source and a “cold” source), or a “light” image and a “dark” image in an optical imaging system. Since two distinct sources are hard to maintain, camera manufacturers use one source to counteract offset drift in real time application, which is often referred to one-point NUC. In a one-point NUC, gain information is stored in a lookup table as a function of temperature, which can be loaded upon update. Given the gain, Equation 1 is solved to obtain the offset b(x,y). Both calibration processes need to interrupt (reset) real time video operations, i.e., a calibration needs to be performed every few minutes to counteract the slow drift of the noise over time and ambient temperature. This is inappropriate for applications such as visual systems used on a battlefield or for video surveillance.
Scene-based NUC techniques have been developed to continuously correct FPA non-uniformity without the need to interrupt the video sequence in real time (reset). These techniques include statistical methods and the registration methods. In certain statistical methods, it is assumed that all possible values of the true-scene pixel are seen at each pixel location, i.e., if a sequence of video images are examined, each pixel is assumed to have experienced a full range of values, say 20 to 220 out of a range of 0 to 255. In general, statistical methods are not computationally expensive, and are easy to implement. But statistical methods generally require many frames and tie camera needs to move in such way as to satisfy the statistical assumption.
Though relatively new, registration-based methods have some desirable features over statistical methods. Registration methods assume that when images are aligned to each other, then aligned images have the same true-scene pixel at a given pixel location. Even if a scene is moving, when a pixel is aligned in all of the images, it will have the same value. Compared to statistical methods, registration methods are much more efficient, requiring fewer frames to recover the original images. However, prior art registration methods which rely on the above assumption can break down when handle significant fix-pattern noise, particularly unstructured fixed pattern noise. The assumption of the same true-scene pixel in the aligned image can also break down when the true signal response is affected by lighting change, automatic gain control (AGC) of the camera, and random noise. Existing methods either assume identical Gaussian fixed-pattern noise or structured pattern noise with known structure.
Moreover, prior art registration methods are reliable for computing restricted types of motion fields, for example, global shift motion (translation). It is desirable for a NUC method to handle parametric motion fields, in particular, affine motion fields, where the images taken by a camera are subjected to translation, rotation, scaling, and shearing. It would also be desirable for a NUC method to enhance the true scene, such as combining several images into a higher resolution images), i.e., a super-resolution image.
Accordingly, what would be desirable, but has not yet been provided, is a NUC method for eliminating fixed pattern noise in imaging systems that can recover clean images as quickly as prior art registration-based methods, can handle unknown structured or non-structured fixed-pattern noise, can work under affine motion shifts, and can improve the quality of recovered images.